Dean Krusienski, Ph.D.

CHS: Small: A Hybrid Brain-Computer Interface for Behaviorally Non-Responsive Patients

NSF

2014-08-01

Brain-computer interfaces (BCIs) have been explored for several years in an effort to provide communication for "locked in" users who have the desire and mental capacity to communicate but are unable to speak, type, or use conventional assistive technologies due to severe motor disabilities, and a lot of work has gone into making this initially crude technology more practical, usable, accurate, and flexible (e.g., by improving speed of performance and providing virtual reality feedback and/or advanced device control). In this project the PI and his team turn their attention to a group with even greater need: patients who have been misdiagnosed as vegetative or minimally conscious, and without the mental ability to form messages or respond to questions. These individuals are not only unable to move but also unable to see, and are at risk of being euthanized based on the mistaken assumption that they are effectively "brain dead" whereas it has been shown by European colleagues, using methods and equipment comparable to American hospitals, that 17-42% of such patients were in fact able to use a BCI to respond to questions. The PI worries that severely injured veterans and others might sometimes be misdiagnosed, potentially able to communicate with friends and loved ones if only some technology could more effectively assess their brain activity. The PI's goal in this project is to extend current BCI technologies to focus on assessing consciousness in this vulnerable patient population and provide, where possible, the ability to communicate.

The PI's approach is to adapt and extend conventional BCI protocols and feedback environments to work with people who cannot see and must instead rely on other modalities of stimulation. The work will involve three thrusts. First, the PI and his team will improve methods to identify brain response based only on tactile and auditory stimuli, by determining the best tactile stimulation frequency for each subject. Second, they will use "hybrid" BCIs combining P300s and steady state somatosensory evoked potentials (SSSEPs) to elicit two different kinds of EEG signals that could improve accuracy. Finally, they will develop a new six choice BCI system tailored for these users; at present the best BCIs for these patient groups allow just two or three choices, whereas a six choice system could lead to faster communication and broader control options. Across all three of these thrusts, the team will also explore signal processing m

Classifier Development forSteady-State Visually Evoked Potential Data

NIA/NASA

2015-09-01

Classifier Development forSteady-State Visually Evoked Potential Data

MRI Consortium: Acquisition of an Integrated System of Instruments for Multichannel Biopotential Recording of In-vitro and In-vivo Experiments

NSF

2013-09-01

An award is made to the Norfolk State University (NSU) to build an integrative research platform for neural recording and analysis and provide the resources for students, faculty and associated collaborators for their research activities from electronic and optical engineering, biology, computer science, nanotechnology, immunology, and neuroscience fields. The addition of this system to the existing research infrastructures will directly benefit and upgrade investigator's research programs at NSU, Eastern Virginia Medical School and Old Dominion University, as well as the collaborative network capabilities in the Hampton Roads area. Applying the multichannel biopotential recording system, neural and cardiac potentials, and neurochemicals such as neurotransmitters can be measured in-vivo and in-vitro, and analyzed by embedded software. The instrument will be applied for the study of neural mechanisms underlying the interactions under various behavioral, stimuli, and disease conditions. In addition to the biopotential recording, the instrumentation will allow in-vivo assessment of neural sensing devices to be developed by engineering research. Furthermore, in-vivo and in-vitro real-time measurement and monitoring of sensing signal can be applied for the development of novel therapeutic treatment strategies of neurological disorders and degeneration.

The outcomes of this synergetic research will be positively transferred to achieve their STEM educational and outreach goals, especially focused for under-represented minority (URM) students in the project?s leading institution, NSU, one of the largest Historically Black Colleges and Universities in the U.S. This project will specifically place a high priority on training of URM students in Hampton Roads region and extend their efforts to URM students nationwide through an aggressive set of research, training and outreach activities utilizing current outreach programs on each campus. The outcomes of this project will be incorporated in their Bioengineering Minor program curriculum. Furthermore, they will also integrate this infrastructure with activity of the newly created Hampton Roads Neuroscience Network.

HCC: Medium: RUI: Control of a Robotic Manipulator via a Brain-Computer Interface

NSF

2009-07-15

A brain-computer interface (BCI) is a system that allows users, especially individuals with severe neuromuscular disorders, to communicate and control devices using their brain waves. There are over two million people in the United States afflicted by such disorders, many of whom could greatly benefit from assistive devices controlled by a BCI. Over the past two years, it has been demonstrated that a non-invasive, scalp-recorded electroencephalography (EEG) based BCI paradigm can be used by a disabled individual for long-term, reliable control of a personal computer. This BCI paradigm allows users to select from a set of symbols presented in a flashing visual matrix by classifying the resulting evoked brain responses. One of the goals of this project is to establish that the same BCI paradigm and techniques used for the aforementioned demonstration can be straightforwardly implemented to generate high-level commands for controlling a robotic manipulator in three dimensions according to user intent, and that such a BCI can provide superior dimensional control over alternative BCI techniques currently available, as well as a wider variety of practical functions for performing everyday tasks.

Electrocorticography (ECoG), electrical activity recorded directly from the surface of the brain, has been demonstrated in recent preliminary work to be another potentially viable control for a BCI. ECoG has been shown to have superior signal-to-noise ratio, and spatial and spectral characteristics, compared to EEG. But the EEG signals used at present to operate BCIs have not been characterized in ECoG. The PI believes ECoG signals can be used to improve the speed and accuracy of BCI applications, including for example control of a robotic manipulator. Thus, additional goals of this project are to characterize evoked responses obtained from ECoG, to use them as control signals to operate a simulated robotic manipulator, and to assess the level of control (speed and accuracy) between the two recording modalities and compare the results to competitive BCI techniques. Because this is a collaborative effort with the Departments of Neurology and Neurosurgery at the Mayo Clinic in Jacksonville, the PI team will have access to a pool of ECoG grid patients from which to recruit participants for this study.

Broader Impacts: This research will make a number of contributions in the emerging field of BCI and thus will serve as a step toward providing severely disabled individuals

Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

C Tremmel, C Herff, T Sato, K Rechowicz,Y Yamani, DJ Krusienski. Estimating Cognitive Workload in an Interactive VR Environment using EEG

2019-11-14

With the recent surge of affordable, high-performance virtual reality (VR) headsets, there is unlimited potential for applications ranging from education, to training, to entertainment, to fitness and beyond. As these interfaces continue to evolve, passive user-state monitoring can play a key role in expanding the immersive VR experience, and tracking activity for user well-being. By recording physiological signals such as the electroencephalogram (EEG) during use of a VR device, the user's interactions in the virtual environment could be adapted in real-time based on the user's cognitive state. Current VR headsets provide a logical, convenient, and unobtrusive framework for mounting EEG sensors. The present study evaluates the feasibility of passively monitoring cognitive workload via EEG while performing a classical n-back task in an interactive VR environment. Data were collected from 15 participants and the spatio-spectral EEG features were analyzed with respect to task performance. The results indicate that scalp measurements of electrical activity can effectively discriminate three workload levels, even after suppression of a co-varying high-frequency activity.

View more

Speech synthesis from ECoG using densely connected 3D convolutional neural networks

M Angrick, C Herff, E Mugler, M Tate, M Slutzky, DJ Krusienski, T Schultz

2019-04-16

Objective. Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach. Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results. In a study with six participants, we achieved correlations up to r  =  0.69 between the reconstructed and original logMel spectrograms. We transferred our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance. To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

View more

Interpretation of convolutional neural networks for speech spectrogram regression from intracranial recordings

M Angrick, C Herff, G Johnson, J Shih, DJ Krusienski, T Schultz

2018-05-19

The direct synthesis of continuously spoken speech from neural activity could provide a fast and natural way of communication for users suffering from speech disorders. Mapping the complex dynamics of neural activity to spectral representations of speech is a demanding task for regression models. Convolutional neural networks have recently shown promise for finding patterns in neural signals and might be a good candidate for this particular regression task. However, the intrinsic agency of the resulting networks is challenging to interpret and thus provides little opportunity to gain insights on neural processes underlying speech. While activation maximization can be used to get a glimpse into what a network has learned for a classification task, it usually does not benefit regression problems.

Here, we show that convolutional neural networks can be used to reconstruct an audible waveform from invasively-measured brain activity. By adapting activation maximization, we present a method that can provide insights from neural networks targeting regression problems. Based on experimental data, we achieve statistically significant correlations between spectrograms of synthesized and original speech. Our interpretation approach shows that trained models reveal that electrodes placed in cortical regions associated with speech production tasks have a large impact on the reconstruction of speech segments.

View more

Biosignal-based Spoken Communication: A Survey

T Schultz, M Wand,T Hueber, DJ Krusienski, C Herff, JS Brumberg

Speech is a complex process involving a wide range of biosignals, including but not limited to acoustics. These biosignals-stemming from the articulators, the articulator muscle activities, the neural pathways, and the brain itself-can be used to circumvent limitations of conventional speech processing in particular, and to gain insights into the process of speech production in general. Research on biosignal-based speech processing is a wide and very active field at the intersection of various disciplines, ranging from engineering, computer science, electronics and machine learning to medicine, neuroscience, physiology, and psychology. Consequently, a variety of methods and approaches have been used to investigate the common goal of creating biosignal-based speech processing devices for communication applications in everyday situations and for speech rehabilitation, as well as gaining a deeper understanding of spoken communication. This paper gives an overview of the various modalities, research approaches, and objectives for biosignal-based spoken communication.

Optimization of Checkerboard Spatial Frequencies for Steady-State Visual Evoked Potential Brain-Computer Interfaces

NR Waytowich, Y Yamani, DJ Krusienski

Steady-state visual evoked potentials (SSVEPs) are oscillations of the electroencephalogram (EEG) which are mainly observed over the occipital area that exhibit a frequency corresponding to a repetitively flashing visual stimulus. SSVEPs have proven to be very consistent and reliable signals for rapid EEG-based brain-computer interface (BCI) control. There is conflicting evidence regarding whether solid or checkerboard-patterned flashing stimuli produce superior BCI performance. Furthermore, the spatial frequency of checkerboard stimuli can be varied for optimal performance. The present study performs an empirical evaluation of performance for a 4-class SSVEP-based BCI when the spatial frequency of the individual checkerboard stimuli is varied over a continuum ranging from a solid background to single-pixel checkerboard patterns. The results indicate that a spatial frequency of 2.4 cycles per degree can maximize the information transfer rate with a reduction in subjective visual irritation compared to lower spatial frequencies. This important finding on stimulus design can lead to improved performance and usability of SSVEP-based BCIs.

Spatio-Temporal Progression of Cortical Activity Related to Continuous Overt and Covert Speech Production in a Reading Task

JS Brumberg, DJ Krusienski, S Chakrabarti, A Gunduz, P Brunner, AL Ritaccio, G Schalk

2016-11-22

How the human brain plans, executes, and monitors continuous and fluent speech has remained largely elusive. For example, previous research has defined the cortical locations most important for different aspects of speech function, but has not yet yielded a definition of the temporal progression of involvement of those locations as speech progresses either overtly or covertly. In this paper, we uncovered the spatio-temporal evolution of neuronal population-level activity related to continuous overt speech, and identified those locations that shared activity characteristics across overt and covert speech. Specifically, we asked subjects to repeat continuous sentences aloud or silently while we recorded electrical signals directly from the surface of the brain (electrocorticography (ECoG)). We then determined the relationship between cortical activity and speech output across different areas of cortex and at sub-second timescales. The results highlight a spatio-temporal progression of cortical involvement in the continuous speech process that initiates utterances in frontal-motor areas and ends with the monitoring of auditory feedback in superior temporal gyrus. Direct comparison of cortical activity related to overt versus covert conditions revealed a common network of brain regions involved in speech that may implement orthographic and phonological processing. Our results provide one of the first characterizations of the spatiotemporal electrophysiological representations of the continuous speech process, and also highlight the common neural substrate of overt and covert speech. These results thereby contribute to a refined understanding of speech functions in the human brain

View more